Represent `%raw`, `%ffi` and `%%raw` with parsetree nodes by cknitt · Pull Request #8879 · rescript-lang/rescript

cknitt · 2026-10-16T14:45:39Z

Part of #8624 ("extension-keyed language features → nodes"). Builds on #8875 (merged).

Summary

%raw, %ffi and %%raw get dedicated parsetree nodes instead of being extensions with a string payload, and Pconst_raw_source is removed.

| Pexp_raw of {kind: raw_kind; name_loc: Location.t; source: raw_source}
| Pstr_raw of {name_loc: Location.t; source: raw_source; attrs: attributes}

and raw_kind = Raw_expression (* %raw *) | Raw_function (* %ffi *)

and raw_source =
  | Raw_string of string loc (* %raw("..."): the JavaScript source *)
  | Raw_template of string loc (* %raw(`...`): the template source as written *)

%raw and %ffi share one node: both lower to the same raw JavaScript, and %ffi only adds the function check and an arity-based type. The kind keeps them apart for printing, typing and the v0 bridge.
The source is kept as written, so the formatter output is unchanged. Ast_raw.js_source gives the JavaScript.
When a node is turned back into an extension (printer, comments table, v0 bridge), a string is printed as a template if its source contains an unescaped ", as the printer did before. Unlike before, the template escapes backquotes, ${ and backslashes where needed, so it parses back to the same JavaScript.
name_loc is the location of %raw / %ffi / %%raw, which cannot be derived from the node's location when attributes precede it.
Payloads the node can't represent (braced or attributed payloads, non-strings) stay extensions and are processed as before.
PPXs still see %raw/%ffi/%%raw extensions with the same v0 encoding, and Ast_mapper_from0 reads them back into the nodes.
The .cmi, .cmt and AST magic numbers are bumped.

Fix

Through a PPX, a template payload such as %raw(`"a\${b}"`) lost its template marker in the v0 bridge. Its escapes were then not decoded, and locations in error messages shifted by one character. The node keeps the template, so compiling with and without a PPX now gives the same result.

Braced string payloads such as %raw({"\"a\""}) were taken as JavaScript source through a PPX, and came back with a @res.braces attribute instead of braces. They are now mapped like any other expression, so they are decoded as ordinary strings with and without a PPX.

Tests

ast-mapping/RawExtensions.res covers strings, templates with escapes, braced strings, %ffi, attributes, doc comments and %%raw.
raw_output_test.res compiles braced %raw and %%raw strings.
ounit_ast_mapper0_tests: raw sources printed as templates parse back to the same JavaScript, and attributed templates survive the v0 bridge.
ounit_ast_mapper0_tests: raw nodes reach v0 as the extensions they are written as and come back unchanged, including a %%raw with attributes.

Compared with master (#8875), the formatter output and the v0 tree a PPX receives are identical for every .res/.resi file in the repository. Compiled output and errors are identical for about 200 %raw/%ffi/%%raw forms in many positions, including types, patterns and modules where they stay extensions. Through a PPX they differ only in the fixes above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NP47qMfjMbjUTKZDsLthDo

`Pexp_raw` (with a `Raw_expression` / `Raw_function` kind) and `Pstr_raw` hold the JavaScript source as written, so `Pconst_raw_source` is gone. PPXs still see the extensions, and v0 templates now keep their decoding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NP47qMfjMbjUTKZDsLthDo Signed-off-by: Christoph Knittel <ck@cca.io>

codecov · 2026-10-16T15:02:35Z

Codecov Report

❌ Patch coverage is 73.83513% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.98%. Comparing base (735a485) to head (36f11f8).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
compiler/ml/printast.ml	0.00%	9 Missing ⚠️
compiler/syntax/src/res_parens.ml	0.00%	8 Missing ⚠️
tests/ounit_tests/ounit_ast_mapper0_tests.ml	87.50%	8 Missing ⚠️
compiler/frontend/ast_exp_handle_external.ml	86.11%	5 Missing ⚠️
compiler/ml/ast_iterator.ml	0.00%	5 Missing ⚠️
compiler/ml/typecore.ml	0.00%	5 Missing ⚠️
compiler/syntax/src/res_comments_table.ml	0.00%	5 Missing ⚠️
compiler/ml/ast_mapper_from0.ml	82.60%	4 Missing ⚠️
compiler/ml/ast_payload.ml	55.55%	4 Missing ⚠️
compiler/ml/ast_raw.ml	91.66%	3 Missing ⚠️
... and 11 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8879      +/-   ##
==========================================
- Coverage   81.01%   80.98%   -0.03%     
==========================================
  Files         454      455       +1     
  Lines       63187    63390     +203     
==========================================
+ Hits        51193    51339     +146     
- Misses      11994    12051      +57

Files with missing lines	Coverage Δ
analysis/src/dump_ast.ml	`44.01% <ø> (+0.18%)`	⬆️
compiler/ext/config.ml	`100.00% <ø> (ø)`
compiler/frontend/bs_builtin_ppx.ml	`90.61% <100.00%> (+0.12%)`	⬆️
compiler/ml/ast_mapper.ml	`98.45% <100.00%> (+0.03%)`	⬆️
compiler/ml/ast_mapper_to0.ml	`77.75% <100.00%> (+0.46%)`	⬆️
compiler/syntax/src/res_core.ml	`91.99% <100.00%> (+<0.01%)`	⬆️
compiler/syntax/src/res_parsetree_viewer.ml	`94.42% <ø> (-0.05%)`	⬇️
compiler/syntax/src/res_printer.ml	`94.03% <100.00%> (+0.10%)`	⬆️
analysis/reanalyze/src/annotation.ml	`52.63% <0.00%> (ø)`
analysis/src/document_symbol.ml	`56.09% <0.00%> (ø)`
... and 19 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cknitt · 2026-10-16T15:04:05Z

@codex review

chatgpt-codex-connector · 2026-10-16T15:04:22Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T19:19:18.581415Z	`36f11f8`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 88b4db6f91

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pkg-pr-new · 2026-10-16T15:13:05Z

Open in StackBlitz

rescript

npm i https://pkg.pr.new/rescript@8879

@rescript/belt

npm i https://pkg.pr.new/@rescript/belt@8879

@rescript/darwin-arm64

npm i https://pkg.pr.new/@rescript/darwin-arm64@8879

@rescript/darwin-x64

npm i https://pkg.pr.new/@rescript/darwin-x64@8879

@rescript/linux-arm64

npm i https://pkg.pr.new/@rescript/linux-arm64@8879

@rescript/linux-x64

npm i https://pkg.pr.new/@rescript/linux-x64@8879

@rescript/runtime

npm i https://pkg.pr.new/@rescript/runtime@8879

@rescript/win32-x64

npm i https://pkg.pr.new/@rescript/win32-x64@8879

commit: 36f11f8

github-actions · 2026-10-16T15:16:30Z

Developer playground preview: https://rescript-lang.github.io/rescript/dev-playground/?version=pr-8879

A braced string payload is an ordinary string, with or without a PPX. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NP47qMfjMbjUTKZDsLthDo Signed-off-by: Christoph Knittel <ck@cca.io>

cknitt · 2026-10-16T15:45:58Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: caa10c51f3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A source that needs a template now escapes backquotes, ${ and backslashes, and attributed templates stay templates through the v0 bridge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NP47qMfjMbjUTKZDsLthDo Signed-off-by: Christoph Knittel <ck@cca.io>

cknitt · 2026-10-16T19:13:25Z

@codex review

github-actions · 2026-10-16T19:13:34Z

Semantic diff: https://cdn.rescript-lang.org/semantic-diff/pr-8879/index.html?v=36f11f847

chatgpt-codex-connector · 2026-10-16T19:19:16Z

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 36f11f847a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cristianoc · 2026-10-17T00:43:27Z

@cknitt, an update on the semantic diff tooling, which affects this PR.

What I missed. #8870 shipped only part of the records work. The object kinds never reached master: parsetree and typed tree drawn as trees, typing rules, the clause matrix, Lambda and JS side by side. Neither did the fact extractor that gets Before/After from real compiler builds, or the commit-msg hook. They sat on a local branch I forgot to merge. #8882 brings them onto current master, with your review applied: no Ran/Depends, no PR citations, Tests required, Signed-off-by last, and fragments as the changelog as in #8881.

Bringing this PR onto the new format. Rebase onto #8882 and rewrite each commit message as a record, following .claude/skills/records/SKILL.md:

One AST modified block on one example, for instance let f = %raw(`(a) => a`) and %%raw("var x = 1"). Extract both trees with scripts/recordFacts.js: Before from a build of the parent commit (the skill gives a worktree recipe), After from this commit's build. Name both builds on the Source: line.
Tests lists every test file the diff changes.
Check the result with make semantic-diff, or with node scripts/commitRecords.js --out page.html <base>..HEAD.

As a trial, an agent re-committed this PR's diff on #8882. It extracted these trees, and every check on its card passes:

Before (parent build) and After (this diff), from bsc -bs-no-builtin-ppx -only-parse -dparsetree

Before:

[
  structure_item
    Pstr_value Nonrec
    [
      <def>
        pattern
          Ppat_var "f"
        expression
          Pexp_extension "raw"
          [
            structure_item
              Pstr_eval
              expression
                Pexp_template
                  source_segment "(a) => a"
          ]
    ]
  structure_item
    Pstr_extension "raw"
    [
      structure_item
        Pstr_eval
        expression
          Pexp_constant PConst_raw_source "var x = 1"
    ]
]

After:

[
  structure_item
    Pstr_value Nonrec
    [
      <def>
        pattern
          Ppat_var "f"
        expression
          Pexp_raw "raw"
          Raw_template "(a) => a"
    ]
  structure_item
    Pstr_raw
    Raw_string "var x = 1"
]

Order. Close this PR, merge #8882, then reopen the %raw nodes work on top of master as a new PR whose commits are records. That way it gets the full semantic diff page from its first push.

@cknitt

Why: `%raw`, `%ffi` and `%%raw` were extensions whose string payload the compiler recognized by name, with a constant `Pconst_raw_source` used only by them; #8624 moves extension-keyed language features to parsetree nodes. Through a PPX, a template payload lost its template marker, so its escapes were not decoded. AST modified (raw extensions): Example: let f = %raw(`(a) => a`) %%raw("var x = 1") Before (parse): [ structure_item Pstr_value Nonrec [ <def> pattern Ppat_var "f" expression Pexp_extension "raw" [ structure_item Pstr_eval expression Pexp_template source_segment "(a) => a" ] ] structure_item Pstr_extension "raw" [ structure_item Pstr_eval expression Pexp_constant PConst_raw_source "var x = 1" ] ] After (parse): [ structure_item Pstr_value Nonrec [ <def> pattern Ppat_var "f" expression Pexp_raw "raw" Raw_template "(a) => a" ] structure_item Pstr_raw Raw_string "var x = 1" ] Before (expanded): [ structure_item Pstr_value Nonrec [ <def> pattern Ppat_var "f" expression attribute "internal.expansive" [] Pexp_letmodule "J" module_expr Pmod_structure [ structure_item Pstr_primitive value_description "unsafe_expr" core_type Ptyp_arrow arity = 1 Nolabel core_type Ptyp_any core_type Ptyp_any "#raw_expr" ] expression Pexp_apply expression Pexp_ident "J.unsafe_expr" [ <arg> Nolabel expression Pexp_constant PConst_raw_source "(a) => a" ] transformed_jsx: false ] structure_item Pstr_eval expression Pexp_letmodule "J" module_expr Pmod_structure [ structure_item Pstr_primitive value_description "unsafe_expr" core_type Ptyp_arrow arity = 1 Nolabel core_type Ptyp_any core_type Ptyp_any "#raw_stmt" ] expression Pexp_apply expression Pexp_ident "J.unsafe_expr" [ <arg> Nolabel expression Pexp_constant PConst_raw_source "var x = 1" ] transformed_jsx: false ] After (expanded): [ structure_item Pstr_value Nonrec [ <def> pattern Ppat_var "f" expression attribute "internal.expansive" [] Pexp_letmodule "J" module_expr Pmod_structure [ structure_item Pstr_primitive value_description "unsafe_expr" core_type Ptyp_arrow arity = 1 Nolabel core_type Ptyp_any core_type Ptyp_any "#raw_expr" ] expression Pexp_apply expression Pexp_ident "J.unsafe_expr" [ <arg> Nolabel expression Pexp_constant PConst_string (source="(a) => a", semantic="(a) => a") ] transformed_jsx: false ] structure_item Pstr_eval expression Pexp_letmodule "J" module_expr Pmod_structure [ structure_item Pstr_primitive value_description "unsafe_expr" core_type Ptyp_arrow arity = 1 Nolabel core_type Ptyp_any core_type Ptyp_any "#raw_stmt" ] expression Pexp_apply expression Pexp_ident "J.unsafe_expr" [ <arg> Nolabel expression Pexp_constant PConst_string (source="var x = 1", semantic="var x = 1") ] transformed_jsx: false ] Review: What does a PPX see? The `%raw`, `%ffi` and `%%raw` extensions with the same v0 encoding as before; `Ast_mapper_from0` reads them back into the nodes. The `.cmi`, `.cmt` and AST magic numbers are bumped. Review: Which payloads stay extensions? Braced or attributed payloads and non-strings, which the nodes cannot represent; they are processed as before. Re-applies #8879 by @cknitt for illustration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cristiano Calcagno <cristianoc@users.noreply.github.com>

cknitt force-pushed the raw-extension-nodes branch from da8d79b to efc319d Compare October 16, 2026 14:45

Base automatically changed from fixed-external-args-parsetree to master October 16, 2026 14:49

cknitt force-pushed the raw-extension-nodes branch from efc319d to 88b4db6 Compare October 16, 2026 14:56

cknitt added this pull request to stack #8880 October 16, 2026 15:02

cknitt mentioned this pull request Oct 16, 2026

Add provisional %raw syntax for fixed external arguments #8876

Draft

cknitt mentioned this pull request Oct 16, 2026

Read the semantic diff's changelog from changes/ fragments #8881

Merged

chatgpt-codex-connector Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread compiler/ml/ast_payload.ml

Decode braced %raw strings as before

caa10c5

A braced string payload is an ordinary string, with or without a PPX. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NP47qMfjMbjUTKZDsLthDo Signed-off-by: Christoph Knittel <ck@cca.io>

chatgpt-codex-connector Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread compiler/ml/ast_mapper_from0.ml

cristianoc mentioned this pull request Oct 17, 2026

Draw commit records as semantic diffs of the compiler's objects #8882

Closed

cristianoc approved these changes Oct 17, 2026

View reviewed changes

cknitt marked this pull request as ready for review October 17, 2026 07:16

cknitt merged commit b3c4184 into master Oct 17, 2026
30 checks passed

cknitt deleted the raw-extension-nodes branch October 17, 2026 07:18

cknitt mentioned this pull request Oct 18, 2026

Stamp parsetree v1: what remains before the AST can be frozen for PPXs #8624

Open

15 of 38 tasks

cristianoc mentioned this pull request Oct 19, 2026

Semantic diff examples (illustration, not for merge) #8909

Draft

Conversation

cknitt commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fix

Tests

Uh oh!

codecov Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cknitt commented Oct 16, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pkg-pr-new Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

cknitt commented Oct 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cknitt commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026

Uh oh!

cristianoc commented Oct 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cknitt commented Oct 16, 2026 •

edited

Loading

codecov Bot commented Oct 16, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 16, 2026 •

edited

Loading

pkg-pr-new Bot commented Oct 16, 2026 •

edited

Loading